{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0UD-1_xY-h2u"
   },
   "source": [
    "# Ungraded Lab: Multi-class Classifier\n",
    "\n",
    "In this lab, you will look at how to build a model to distinguish between more than two classes. The code will be similar to the ones you've been using before with a few key changes in the model and in the training parameters. Let's dive in!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import numpy as np\n",
    "from io import BytesIO\n",
    "\n",
    "# Plotting and dealing with images\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# Interactive widgets\n",
    "from ipywidgets import widgets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FvwVR5lHA8q_"
   },
   "source": [
    "## Inspect the Dataset\n",
    "\n",
    "You will be using the [Rock-Paper-Scissors dataset](https://www.tensorflow.org/datasets/catalog/rock_paper_scissors), a gallery of hands images in Rock, Paper, and Scissors poses.\n",
    "\n",
    "As usual, you will assign the directory names into variables and look at the filenames as a sanity check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MrxdR83ANgjS"
   },
   "outputs": [],
   "source": [
    "BASE_DIR = './rps'\n",
    "\n",
    "rock_dir = os.path.join(BASE_DIR, 'rock')\n",
    "paper_dir = os.path.join(BASE_DIR, 'paper')\n",
    "scissors_dir = os.path.join(BASE_DIR, 'scissors')\n",
    "\n",
    "print(f'total training rock images: {len(os.listdir(rock_dir))}')\n",
    "print(f'total training paper images: {len(os.listdir(paper_dir))}')\n",
    "print(f'total training scissors images: {len(os.listdir(scissors_dir))}')\n",
    "\n",
    "rock_files = os.listdir(rock_dir)\n",
    "paper_files = os.listdir(paper_dir)\n",
    "scissors_files = os.listdir(scissors_dir)\n",
    "\n",
    "print()\n",
    "print(f\"5 files in the rock subdir: {rock_files[:5]}\")\n",
    "print(f\"5 files in the paper subdir: {paper_files[:5]}\")\n",
    "print(f\"5 files in the scissors subdir: {scissors_files[:5]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7t_CNSs6B-8y"
   },
   "source": [
    "You can also inspect some of the images to see the variety in your model inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jp9dLel9N9DS"
   },
   "outputs": [],
   "source": [
    "next_rock = [os.path.join(rock_dir, fname)\n",
    "             for fname in random.sample(rock_files, k=2)]\n",
    "next_paper = [os.path.join(paper_dir, fname)\n",
    "              for fname in random.sample(paper_files, k=2)]\n",
    "next_scissors = [os.path.join(scissors_dir, fname)\n",
    "                 for fname in random.sample(scissors_files, k=2)]\n",
    "\n",
    "for i, img_path in enumerate(next_rock+next_paper+next_scissors):\n",
    "    img = mpimg.imread(img_path)\n",
    "    plt.imshow(img)\n",
    "    plt.axis('Off')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ufa0YF5oCpYw"
   },
   "source": [
    "## Build the model\n",
    "\n",
    "You will then build your CNN. You will use 4 convolution layers with 64-64-128-128 filters then append a `Dropout` layer to avoid overfitting and some `Dense` layers for the classification. The output layer would be a 3-neuron `Dense` layer activated by [Softmax](https://www.tensorflow.org/api_docs/python/tf/nn/softmax). You've seen this in Course 1 when you were training with Fashion MNIST. It scales your output to a set of probabilities that add up to 1. The order of this 3-neuron output would be paper-rock-scissors (e.g. a `[0.8 0.2 0.0]` output means the model is predicting 80% probability for paper and 20% probability for rock.\n",
    "\n",
    "You can examine the architecture with `model.summary()` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NrJEfhfEqlcJ"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.Input(shape=(150, 150, 3)),\n",
    "    # Rescale the image. Note the input shape is the desired size of the image: 150x150 with 3 bytes for color\n",
    "    tf.keras.layers.Rescaling(1./255),\n",
    "    # This is the first convolution\n",
    "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2, 2),\n",
    "    # The second convolution\n",
    "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2,2),\n",
    "    # The third convolution\n",
    "    tf.keras.layers.Conv2D(128, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2,2),\n",
    "    # The fourth convolution\n",
    "    tf.keras.layers.Conv2D(128, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2,2),\n",
    "    # Flatten the results to feed into a DNN\n",
    "    tf.keras.layers.Flatten(),\n",
    "    tf.keras.layers.Dropout(0.5),\n",
    "    # 512 neuron hidden layer\n",
    "    tf.keras.layers.Dense(512, activation='relu'),\n",
    "    tf.keras.layers.Dense(3, activation='softmax')\n",
    "])\n",
    "\n",
    "# Print the model summary\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7Ps7kIRaFRIC"
   },
   "source": [
    "## Preprocess the Image Data\n",
    "\n",
    "You will prepare the training and validation datasets as before. The label mode will be `categorical` because you will predict more than two classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Au_DSdZ5pxRb"
   },
   "outputs": [],
   "source": [
    "TRAINING_DIR = \"./rps\"\n",
    "VALIDATION_DIR = \"./rps-test-set\"\n",
    "\n",
    "# Instantiate the training dataset\n",
    "train_dataset = tf.keras.utils.image_dataset_from_directory(\n",
    "    TRAINING_DIR,\n",
    "    image_size=(150, 150),\n",
    "    batch_size=32,\n",
    "    label_mode='categorical'\n",
    "    )\n",
    "\n",
    "# Instantiate the validation dataset\n",
    "validation_dataset = tf.keras.utils.image_dataset_from_directory(\n",
    "    VALIDATION_DIR,\n",
    "    image_size=(150, 150),\n",
    "    batch_size=32,\n",
    "    label_mode='categorical'\n",
    "    )\n",
    "\n",
    "# Optimize the datasets for training\n",
    "SHUFFLE_BUFFER_SIZE = 1000\n",
    "PREFETCH_BUFFER_SIZE = tf.data.AUTOTUNE\n",
    "\n",
    "train_dataset_final = (train_dataset\n",
    "                       .cache()\n",
    "                       .shuffle(SHUFFLE_BUFFER_SIZE)\n",
    "                       .prefetch(PREFETCH_BUFFER_SIZE)\n",
    "                       )\n",
    "\n",
    "validation_dataset_final = (validation_dataset\n",
    "                            .cache()\n",
    "                            .prefetch(PREFETCH_BUFFER_SIZE)\n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YdTv_jIuygwu"
   },
   "source": [
    "## Prepare the Model for Training\n",
    "\n",
    "You will use data augmentation to generate other poses that the model needs to learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qK-fS5koqcDX"
   },
   "outputs": [],
   "source": [
    "# Create a model with data augmentation layers\n",
    "data_augmentation = tf.keras.Sequential([\n",
    "    tf.keras.Input(shape=(150, 150, 3)),\n",
    "    tf.keras.layers.RandomFlip(\"horizontal\"),\n",
    "    tf.keras.layers.RandomRotation(0.4),\n",
    "    tf.keras.layers.RandomTranslation(0.2,0.2),\n",
    "    tf.keras.layers.RandomContrast(0.4),\n",
    "    tf.keras.layers.RandomZoom(0.2)\n",
    "    ])\n",
    "\n",
    "# Attach the data augmentation model to the base model\n",
    "model_with_aug = tf.keras.models.Sequential([\n",
    "    data_augmentation,\n",
    "    model\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0zTD9SgDz-cC"
   },
   "source": [
    "You will compile the model using a [`categorical_crossentropy`](https://www.tensorflow.org/api_docs/python/tf/keras/losses/CategoricalCrossentropy) loss function to quantify the error across all 3 classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HY4G-rFVrCD2"
   },
   "outputs": [],
   "source": [
    "# Set the training parameters\n",
    "model_with_aug.compile(loss = 'categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Orf1QQlGGyOe"
   },
   "source": [
    "## Train the model and evaluate the results\n",
    "\n",
    "You will train for 25 epochs and evaludate the results afterwards. Observe how both the training and validation accuracy are trending upwards. This is a good indication that the model is not overfitting to the training set.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1mHX5L7HFXQ7"
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "history = model_with_aug.fit(\n",
    "    train_dataset_final,\n",
    "    epochs=25,\n",
    "    validation_data = validation_dataset_final,\n",
    "    verbose = 1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aeTRVCr6aosw"
   },
   "outputs": [],
   "source": [
    "def plot_loss_acc(history):\n",
    "  '''Plots the training and validation loss and accuracy from a history object'''\n",
    "  acc = history.history['accuracy']\n",
    "  val_acc = history.history['val_accuracy']\n",
    "  loss = history.history['loss']\n",
    "  val_loss = history.history['val_loss']\n",
    "\n",
    "  epochs = range(len(acc))\n",
    "\n",
    "  fig, ax = plt.subplots(1,2, figsize=(12, 6))\n",
    "  ax[0].plot(epochs, acc, 'bo', label='Training accuracy')\n",
    "  ax[0].plot(epochs, val_acc, 'b', label='Validation accuracy')\n",
    "  ax[0].set_title('Training and validation accuracy')\n",
    "  ax[0].set_xlabel('epochs')\n",
    "  ax[0].set_ylabel('accuracy')\n",
    "  ax[0].legend()\n",
    "\n",
    "  ax[1].plot(epochs, loss, 'bo', label='Training Loss')\n",
    "  ax[1].plot(epochs, val_loss, 'b', label='Validation Loss')\n",
    "  ax[1].set_title('Training and validation loss')\n",
    "  ax[1].set_xlabel('epochs')\n",
    "  ax[1].set_ylabel('loss')\n",
    "  ax[1].legend()\n",
    "\n",
    "  plt.show()\n",
    "\n",
    "plot_loss_acc(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y3ps8Q1tpYMG"
   },
   "source": [
    "# Model Prediction\n",
    "\n",
    "You can feed in a picture and have the model classify it as rock, paper, or scissors. You can upload your own images or use the ones available [here](https://storage.googleapis.com/tensorflow-1-public/course2/week4/rps-validation.zip)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZABJp7T3VLCU"
   },
   "outputs": [],
   "source": [
    "# Create the widget and take care of the display\n",
    "uploader = widgets.FileUpload(accept=\"image/*\", multiple=True)\n",
    "display(uploader)\n",
    "out = widgets.Output()\n",
    "display(out)\n",
    "\n",
    "def file_predict(filename, file, out):\n",
    "    \"\"\" A function for creating the prediction and printing the output.\"\"\"\n",
    "    image = tf.keras.utils.load_img(file, target_size=(150, 150))\n",
    "    image = tf.keras.utils.img_to_array(image)\n",
    "    image = np.expand_dims(image, axis=0)\n",
    "\n",
    "    prediction = model.predict(image, verbose=0)[0]\n",
    "\n",
    "    with out:\n",
    "\n",
    "        print(f'\\nmodel output: {prediction}')\n",
    "        \n",
    "        prediction_index = np.argmax(prediction)\n",
    "        \n",
    "        classes = [\"paper\", \"rock\", \"scissors\"]\n",
    "        \n",
    "        predicted_class = classes[prediction_index]\n",
    "        \n",
    "        print(f'{filename} is {predicted_class}')\n",
    "\n",
    "\n",
    "def on_upload_change(change):\n",
    "    \"\"\" A function for geting files from the widget and running the prediction.\"\"\"\n",
    "    # Get the newly uploaded file(s)\n",
    "    \n",
    "    items = change.new\n",
    "    for item in items: # Loop if there is more than one file uploaded  \n",
    "        file_jpgdata = BytesIO(item.content)\n",
    "        file_predict(item.name, file_jpgdata, out)\n",
    "\n",
    "\n",
    "uploader.observe(on_upload_change, names='value')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JHRufhQYJJLU"
   },
   "source": [
    "## Wrap Up\n",
    "\n",
    "That concludes this short exercise on the multi-class classifiers. You saw that with just a few changes, you were able to convert your binary classifiers to predict more classes. You used the same techniques for data and model preparation and were able to get relatively good results in just 25 epochs. For practice, you can search for other datasets (e.g. [here](https://archive.ics.uci.edu/datasets)) with more classes and revise the model to accomodate it. Try to experiment with different layers and data augmentation techniques to improve your metrics."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "private_outputs": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
